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Abstract 

Background: In the last decade, guideline organizations faced a number of problems, including a lack of 
standardization in guideline development methods and suboptimal guideline implementation. To contribute to 
the solution of these problems, we produced a toolbox for guideline development, implementation, revision, and 
evaluation. 

Methods: All relevant guideline organizations in the Netherlands were approached to prioritize the topics. We sent 
out a questionnaire and discussed the results at an invitational conference. Based on consensus, twelve topics were 
selected for the development of new tools. Subsequently, working groups were composed for the development of 
the tools. After development of the tools, their draft versions were pilot tested in 40 guideline projects. Based on 
the results of the pilot tests, the tools were refined and their final versions were presented. 

Results: The vast majority of organizations involved in pilot testing of the tools reported satisfaction with using the 
tools. Guideline experts involved in pilot testing of the tools proposed a variety of suggestions for the implementation 
of the tools. The tools are available in Dutch and in English at a web-based platform on guideline development and 
implementation (www.ha-ring.nl). 

Conclusions: A collaborative approach was used for the development and evaluation of a toolbox for development, 
implementation, revision, and evaluation of guidelines. This approach yielded a potentially powerful toolbox for 
improving the quality and implementation of Dutch clinical guidelines. Collaboration between guideline organizations 
within this project led to stronger linkages, which is useful for enhancing coordination of guideline development and 
implementation and preventing duplication of efforts. Use of the toolbox could improve quality standards in the 
Netherlands, and might facilitate the development of high-quality guidelines in other countries as well. 
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Background 

Since the 1980s, Dutch healthcare professionals have 
invested much time and resources on development and 
implementation of hundreds of guidelines [1]. In the 
1990s, the Dutch Institute for Healthcare Improvement 
CBO and the Dutch Cochrane Center took a next step 
aiming to improve the quality of guideline development 
by starting a national collaboration among guideline de- 
velopers. This resulted in a platform involving almost 30 
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organizations [2]. Currently, almost all those professional 
organizations are actively developing guidelines. This is 
encouraged by governmental legislation and regulation, 
aiming to increase transparency in healthcare quality and 
costs. 

In the last decade, guideline organizations faced new 
problems [1]. First, both guideline developers and imple- 
mentation experts reported a lack of methodological 
support in producing high-quality guidelines and felt a 
need for specific tools to help them in improving the 
quality of the guidelines [3]. A second problem is that 
the development and updating of high-quality guidelines 
require substantial resources, while government is cutting 



o 



Bio Med Central 



© 2014 Hilbink et al.; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative 
Commons Attribution License (http//creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and 
reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain 
Dedication waiver (httpy/creati vecommons.org/publ icdomain/zero/1.0/) applies to the data made available in this article, 
unless otherwise stated. 



Hilbink et al. Implementation Science 2014, 9:34 
http://www.implementationscience.eom/content/9/1/34 



Page 2 of 7 



back spending on guideline development. Therefore, most 
organizations are under pressure to produce more guide- 
lines in a shorter time with the same quality. Addressing 
the views and perspectives of patients and the public, for 
instance, is mandatory but often complex and time- 
consuming [4]. Third, national guideline developers do 
not always feel responsible for implementation and may 
leave it to regional or local groups. Effective implemen- 
tation, however, often requires a multifaceted national 
approach as wider national initiatives generally have a 
more powerful influence than local implementation ef- 
forts [5] . However, such a national approach is not always 
feasible: in locally heterogeneous conditions, for example, 
implementation activities must be tailored to local bar- 
riers and enablers. In the Netherlands, the implementa- 
tion of guidelines lacks a well-developed infrastructure 
and sustainable programs [6] . 

In order to face these problems, the Dutch Government 
established a Quality Institute in 2012 as an independ- 
ent body with the aim to improve the quality of Dutch 
healthcare. The Dutch Quality Institute is convinced 
that patients, healthcare professionals, and healthcare 
insurers are experts in defining high-quality healthcare, 
and that they themselves have to make agreements on 
quality standards. Those quality standards are central in 
supporting the government's vision for a health and so- 
cial care system focused on delivering the best possible 
outcomes for patients and civilians. The role and pos- 
ition of the Dutch Quality Institute is comparable to 
the National Institute for Health and Care Excellence 
(NICE) in the provision of guidance and advice to im- 
prove health and social care in the UK. 

In order to fulfill these expectations, quality criteria 
were formulated for initiating, developing, and imple- 
menting guidelines in the Netherlands [7]. In addition, 
tools could be helpful to meet these quality standards. 
The objective of this paper is to describe the process of 
the development and evaluation of a toolbox for devel- 
opment, implementation, revision, and evaluation of 
guidelines in order to improve quality standards in the 
Netherlands and to be helpful for other countries. 

Methods 

Topic selection 

Based on an analysis of barriers to the development and 
implementation of guidelines [8], a long list of potential 
topics for tools was composed. For each potential topic, 
we searched the literature for existing supportive mate- 
rials. To prioritize the most relevant topics, we con- 
ducted a web-based survey among national guideline 
developing organizations and opinion leaders in the field 
of guideline development and implementation (Table 1). 
One hundred and forty-three online questionnaires were 
sent out and 74 questionnaires were returned, yielding a 



response rate of 52%. Responders were fairly equally di- 
vided between the approached professional groups. For 
each topic, the questionnaire included questions on 
awareness of existing guidance or tools; use of these ma- 
terials in daily practice; and the extent to which there 
was a need for new supportive materials or tools. A 
nine-point response scale was used with answers ranging 
from 'strong need' to 'no need.' They were also asked to 
provide a top-three ranking of most relevant topics for 
the development of new tools. Finally, participants were 
asked to suggest additional topics that were not included 
in the overview of potential topics. In September 2010, 
we organized a national invitational conference to reach 
consensus on the topics for new tools. At this conference 
(attended by 49 responders to the online questionnaire), 
the results of the survey were presented. Problems as well 
as possible solutions on topics that scored a strong need 
for new tools were subsequently discussed in small groups. 
Finally, consensus was reached about a top ten of topics, 
added with two topics preselected by the funding 
organization of our project (cost-effectiveness in guide- 
lines, shared decision making in guidelines). Table 2 pro- 
vides an overview of the topics that were selected for the 
development of new tools. Table 2 shows that the se- 
lected topics cover the entire guideline development pro- 
ject: three topics specifically relate to the preparation 
phase of guideline development, three topics pertain to 
the development phase, and four topics refer to the com- 
pletion phase. Two topics cover the entire guideline 
development process, i.e., organization and cooperation 
in multidisciplinary guideline development and project 
management in guideline development. 

Development of tools 

In October 2010, all Dutch organizations involved in 
guideline development, revision, implementation, and 
monitoring (n = 24) were asked to delegate professionals 
with specific expertise on one or more selected topics 
for tools. Thirteen working groups, each consisting of 
four or five people from different organizations (Table 1), 
were composed for the development of the tools. Each 
working group appointed a chairman, met four to six 
times face-to-face, and arranged some additional con- 
tacts by e-mail, telephone, or video-conferencing. After 
six months, each group delivered a draft version of the 
tool. This draft tool was sent to all organizations involved 
and a large group of experts for comment and review. One 
hundred and fifty experts submitted valuable suggestions 
for improvement. The received comments were analyzed 
and integrated in a next version of the tool. 

Pilot testing of tools 

National organizations involved in the development 
and implementation of clinical practice guidelines were 
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Table 1 Organizations involved in the topic selection, tools development and/or pilot testing of the tools 



Topic selection 



Tools development 



Pilot testing 



Professional society 

Dutch Order of Medical Specialists (OMS) 
Dutch College of General Practitioners (NHG) 
Royal Dutch Society for Physical Therapy (KNGF) 
Dutch Association of Nurses and Care Providers (V&VN) 
Dutch Association of Youth Health Care Doctors (AJN) 
Dutch Urological Association (NVU) 
Research institute 

Netherlands Organisation for Applied Scientific Research (TNO) 

Netherlands institute for health services research (NIVEL) 

institute for Medical Technology Assessment (iMTA) 

Quality improvement organization 

Dutch Institute for Healthcare Improvement (CBO) 

Dutch Council for Quality of Healthcare 

Knowledge institute 

Trimbos Institute 

Dutch Youth Health Centre (NCJ) 

Netherlands Youth Institute (NJi) 

Medical Centre 

Academic Medical Center (AMC) 

Onze Lieve Vrouwe Gasthuis (OLVG) 

Radboud University Nijmegen Medical Centre (RUNMC) 

University Medical Center Groningen (UMCG) 

Leiden University Medical Center (LUMC) 

University Medical Center Utrecht 

Educational organizations 

Maastricht University 

Utrecht University 

Disease specific organization 

Comprehensive Cancer Centre The Netherlands (IKNL) 

Dutch Lung Alliance (LAN) 

Other 

PROVA 

Federation of Patients and Consumers Organisations in the Netherlands (NPCF) 
Dutch Burns Foundation 



approached to test the tools. Considering their current 
and future guideline projects, representatives of those or- 
ganizations selected specific guideline projects for pilot 
testing of the tools. In 40 guideline projects selected, one 
or more tools were tested. In total, 18 organizations were 
involved in the pilot testing of the tools (Table 1). At the 
start of the evaluation, the project leader and chairman 
of the guideline group received detailed information on 
both the content of the tools and the aims of the pilot 



test. Halfway through the evaluation process, the project 
leader of the guideline project was contacted by tele- 
phone to inquire about the status of the evaluation of the 
tools. At the end of the evaluation process of the tools, a 
one-hour face-to-face meeting with the project leader, 
chairman and one or two other members of the guideline 
group was arranged. During this meeting, the following 
topics were discussed: experiences with the use of the 
tool and intentions to use the tools in future guideline 
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Table 2 Topics selected for the development of new tools 


Topic 


Phase* 


1. Analysis of barriers in clinical care 


P 


2. Cost-effectiveness in guidelines 


P 


3. Prevention of improper influence due to conflicts of interest 


P 


4. International collaboration in guideline development 


D 


5. Developing popu ation and sex specific recommendations 


u 


6. Shared decision making in guidelines 


n 

u 


7. Knowledge gaps in guidelines 


c 


8. Implementation of guidelines 


c 


9. Monitoring of guidelines 


c 


10. Electronic disclosure of guidelines 


c 


1 1. Organization and cooperation in multidisciplinary guideline 


P, D, C 


development 




12. Project management in guideline development. 


P, D, C 


*Phase of the guideline development process to which the topic pertains. 
P: preparation phase; D: development phase; C: completion phase. 



projects; suggestions for improvement of the tool; and 
suggestions for dissemination and implementation of the 
tool. Based on the information yielded in these meetings, 
the tools were revised and their final versions were pro- 
duced. With the exception of the tool on the prevention 
of improper influence due to conflicts of interest, the 
pilot testings of the tools were qualitative evaluations. In 
order to evaluate the conflicts of interests tool, individual 
members of guideline projects were approached with 
the request to complete a web-based questionnaire 
on their experiences and satisfaction with completing the 
'Declaration of Interests' form. The participants answered 
the questions by marking their responses on a five-point 
Likert Scale. 

Results 

Experiences with the use of the tools and intention to use 
the tools in future guideline projects 

The vast majority of experts involved in pilot testing of 
the tools reported satisfactory experiences with the use 
of the tools. The tools provided them the necessary sup- 
port to deal with particular aspects of the guideline de- 
velopment, revision, implementation, and monitoring 
process in an improved manner. In particular, tools with 
a more practical structure, such as checklists and step- 
by-step plans, received positive feedback during the pilot 
test. Two tools covering the entire guideline process, i.e., 
the tools on process management and on project man- 
agement, were perceived as too broad in scope. Several 
experts experienced the working methods as described 
in the tools as more time-consuming compared to their 
routine procedures. Furthermore, some tools contain ra- 
ther innovative approaches that differ significantly from 
common methods of guideline development. For example, 



the tool for the analysis of barriers in clinical care regards 
the stages of identifying, analyzing, and ultimately selecting 
barriers in clinical care as an independent process that oc- 
curs before any planned guideline project. Until now, how- 
ever, the analysis of barriers in clinical care has been part 
of the guideline development process. 

The intention to use the tools in future guideline pro- 
jects varied among the participating experts and depended 
on a number of factors, such as the availability and satis- 
faction with current procedures, the experience of key 
figures in the guideline development group and the time 
needed to apply the tool. Most experts involved in pilot 
testing of the tools had the intention to use one or more 
tools in future guideline projects, or at least specific parts 
of the tools that appeared to be most helpful for them. 

Suggestions for improvement of the tools 

After pilot testing of the tools, all tools underwent revi- 
sions on their content. Some tools were only slightly 
adapted to clarify their content, whereas a few tools were 
revised thoroughly based on the experiences during the 
pilot test. The scope of the tools on process manage- 
ment and project management was narrowed: the con- 
tent of the tools was restricted to guideline projects, and 
general information on process and project management 
was removed. The tool on implementation of guidelines 
was aligned with the model for effective implementation 
of Grol etal. [9]. 

Suggestions for dissemination and implementation 
of the tools 

Experts involved in pilot testing of the tools proposed 
the following suggestions for dissemination and imple- 
mentation of the tools. 

1. Design of a digital knowledge platform on guideline 
development and implementation for dissemination 
of the tools; 

2. Publication of the tools in national and international 
scientific journals; 

3. Inclusion of the tools in educational materials and 
educational meetings for guideline developers and 
implementation experts; 

4. Inclusion of use of the tools in funding requirements 
for grants on guideline development and 
implementation; 

5. Availability of sufficient resources and financial support 
to develop, revise, implement, and monitor guidelines 
according to the approach suggested in the tools. 

Evaluation of the tool on the prevention of improper 
influence due to conflicts of interest 

The questionnaire was sent to 112 members of eight dif- 
ferent guideline projects. From the approached members, 
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Table 3 Main characteristics of the developed tools 



Title of the tool 



Objective(s) of the tool 



Structure of the tool 



Analysis of barriers in clinical care 
(www.ha-ring.nl/en/tool-1) 



ntegration of cost-effectiveness in 
guidelines (www.ha-ring.nl/en/tool-2) 



Prevention of improper influence due to 
conflicts of interest (www.ha-ring.nl/en/tool-4) 



International collaboration in guideline 
development (www.ha-ring.nl/en/tool-5) 



Developing population-specific 
recommendations (www.ha-ring.nl/en/tool-7) 



ntegration of shared decision making in 
guidelines (www.ha-ring.nl/en/tool-8) 



Definition and identification of knowledge 
gaps in guidelines (www.ha-ring.nl/en/tool-9) 



implementation of guidelines 
(www.ha-ring.nl/en/tool-1 0) 



Monitoring of guidelines 
(www.ha-ring.nl/en/tool-1 1) 

Electronic disclosure of guidelines 
(www.ha-ring.nl/en/tool-12) 



Organization and cooperation in multidisciplinary 
guideline development (www.ha-ring.nl/en/tool-3) 



Project management in guideline development 
(www.ha-ring.nl/en/tool-6) 



To support the processes of 1) defining the 
scope of the guideline; 2) identifying barriers 
in clinical care; 3) analyzing barriers in clinica 
care; and 4) selecting and prioritizing barriers 
in clinical care 

1) To demonstrate how cost-effectiveness 
and/or budget impact can be involved in 
drafting a guideline 

2) To offer support in evaluating whether 
additional analyses are needed and what 
type of expertise this will require 

To enlarge transparency about relationships 
and interests of all proposed stakeholders 
involved in preparing scientific advisory 
reports and medical guidelines 

To formulate both conditions for successful 
international collaboration and steps that 
guideline developers need to follow when 
collaborating at an international level 

To facilitate the development of 
recommendations that are as specific 
as possible 

To promote shared decision making between 
health care providers and patients when 
implementing preference-sensitive guideline 
recommendations in practice 

1) To facilitate both the identification and 
definition of knowledge gaps during the 
guideline development process 

2) To support researchers and grant providers 
in selecting and funding themes that have a 
clear relevance to patient care 

To stress the importance of prompt attention 
for implementation of the guideline, with the 
ultimate aim of improving the implementability 
of the guideline 

To provide a useful framework for the 
development of a monitoring plan 

To improve the ability to search and find 
guidelines 



To provide practical tips for, and examples 
of, process management, to promote 
(interprofessional) cooperation in guideline 
development groups 

To provide tips, instruments and helpful tools 
for efficient project management when 
developing or revising guidelines and/or 
products derived from them 



Six-step plan for determining, selecting, 
and addressing barriers in clinical care 



Descriptive text, including taxonomy, 
general information and recommendations 



'Declaration of Interests' form 



Six-step plan on how to collaborate 
internationally in developing guidelines 



Checklist, containing a number of 
questions that a guideline development 
group can ask when formulating 
recommendations 

Descriptive text using hyperlinks to 
background materials and papers 



Descriptive text, including a definition 
of knowledge gaps, information on 
forms of knowledge gaps and criteria 
for knowledge gaps 



Checklist, containing a number of points 
that a guideline development group 
can consider, which will facilitate the 
implementation of the guideline 
recommendations in practice 

Eight-step plan for the construction of 
a monitoring plan 

Overview of available Dutch guideline 
databases; checklist of criteria for 
improving the ability to search and find 
guidelines 

Descriptive text, including general 
information, recommendations and 
examples 

Descriptive text, including tips, tricks 
and helpful tools for efficient project 
management 



50 people (45%) filled in a questionnaire. Ninety percent of 
the respondents answered that the various relations and in- 
terests mentioned in the 'Declaration of Interests' form 
were completely clear for them. Nearly all respondents 
(94%) considered the form as an exhaustive and 



comprehensive document that would not miss any rela- 
tions or interests. The respondents had differing points of 
view on the contribution of the 'Declaration of Interests' 
form to the actual prevention of improper influence. About 
one-half of the respondents (56%) answered that the form 
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will enlarge both the awareness and the transparency on 
improper influence due to conflicts of interests, but consid- 
ered this as insufficient for the effective prevention of it. Fi- 
nally, ten respondents (9%) answered that the extent to 
which improper influence due to conflicts of interests oc- 
curs is not measurable with this form. Nevertheless, the 
large majority of respondents (85%) considered completion 
of the form as a first important step towards transparency 
and clarity on possible relations and interests and a useful 
document for assessment of the risk for improper influence 
due to conflicts of interest. 

The final toolbox 

Twelve tools on guideline development, implementation, 
revision, and monitoring were produced, according to 
the following format: title of the tool; background; 
objective of the tool; content of the tool; and relevant lit- 
erature. The tools differ in terms of structure: some tools 
include a step-by-step plan or a checklist, whereas other 
tools have a more descriptive character. Table 3 summa- 
rizes the main characteristics of the tools, i.e., the title of 
the tool, the objective(s) of the tool, and the structure 
of the tool. The full-text versions of the tools are 
available in Dutch and in English via a comprehensive 
website on guideline development and implementation 
(www.ha-ring.nl) . 

Discussion 

Our project showed that national collaboration among 
Dutch guideline organizations resulted in a toolbox cov- 
ering a set of relevant topics on guideline methodology. 
Experts involved in pilot testing of the tools reported 
satisfactory experiences with the use of the tools. A 
recent comparative review of clinical practice guideline 
development handbooks showed that specific support 
for evidence based development of guidelines was lack- 
ing [3]. This toolbox can have added value to existing 
supportive materials on guideline development and 
implementation [10-12], as it elaborates on various 
aspects of the guideline development process for 
which there appeared to be a great need for new supportive 
materials. Because the toolbox covers the complete 
guideline development project (preparation phase, 
development phase, and completion phase), it could 
have a major impact on many aspects of the guideline 
development process in The Netherlands and in other 
countries as well. 

The tools are available via a comprehensive website 
on guideline development and implementation (www. 
ha-ring.nl). This digital knowledge platform could be a 
powerful instrument for improving Dutch clinical 
guidelines. The recently established Dutch Quality 
Institute will decide about proposed additions to the 
knowledge platform and will keep the platform up-to-date. 



Furthermore, the Dutch Quality Institute is currendy 
developing a national registry and database of clinical 
practice guidelines using the National Clearinghouse 
Guidelines and NHS Evidence as examples. The Institute 
has recommended the use of our tools in order to 
fulfill the quality criteria for inclusion in the database. 
This toolbox facilitates the Dutch Quality Institute in 
improving the quality of healthcare and in providing 
guidance and advice to improve healthcare as NICE 
does in the UK. 

Several limitations of our work need to be mentioned. 
First, all tools are based on experts' opinion on best 
practices of producing and implementing guidelines. Al- 
though everyone involved in the development and pilot 
testing of the tools are leading experts on the concerning 
topics, certain comments of other experts contained 
suggestions for a somewhat different approach. Second, 
it might be difficult to encourage persons who are highly 
experienced in the field of guideline development and 
implementation to use these tools, because they possibly 
prefer to stick to their own methods and routines. Hope- 
fully, their less experienced colleagues, to whom the 
tools will be disseminated via educational materials and 
meetings, will succeed in convincing them of the value 
of these tools. Finally, the tools on implementation 
might have limited international generalizability, because 
countries that are larger and have a more heterogeneous 
legislation than the Netherlands probably benefit more 
from an implementation approach tailored to the local 
context. 

Beyond the development of the toolbox, our study also 
led to stronger linkages between Dutch organizations in- 
volved in guideline development and implementation. As 
a consequence, they established the Collaborative Dutch 
Association for Excellent Guidelines, called GENEVER, in 
2012. The establishment of GENEVER, a paraphrase to a 
well-known Dutch liquor, has resulted in an intensive ex- 
change of knowledge and information among its mem- 
bers, similar as in the Guidelines International Network 
(www.g-i-n.net). All members of GENEVER are familiar 
with the tools and willing to support the dissemination of 
the toolbox and its implementation within the national 
and international context. Implementation of the tools 
will facilitate the development of high-quality guide- 
lines, which could result in higher guideline adherence 
and, ultimately, in improved quality of care. 

Conclusions 

A collaborative approach was used for the development 
and evaluation of a toolbox for development, imple- 
mentation, revision, and evaluation of guidelines. This 
approach yielded a potentially powerful toolbox for im- 
proving the quality and implementation of Dutch clinical 
guidelines. Collaboration between guideline organizations 
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within this project led to stronger linkages, which is 
useful for enhancing coordination of guideline develop- 
ment and implementation and preventing duplication 
of efforts. Use of the toolbox could improve quality 
standards in the Netherlands, and might facilitate the 
development of high-quality guidelines in other countries 
as well. 
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